Optical overlay device

ABSTRACT

Devices and methods for visibly highlighting areas of a region including an imager configured to image the region with a sensitivity to at least one of wavelength, light level, or contrast greater than the human eye, an overlay element configured to visibly highlight areas of the region and registered to the imager to produce alignment of imaged features with highlighted features at the same location on the region, and at least one of a controller executing a program or logic configured to process acquired images from the imager to identify areas of the region determined not visible to the human eye, and control the overlay element to visibly highlight those areas on the region.

This application is continuation of U.S. application Ser. No. 16/752,640filed Jan. 25, 2020; which is a continuation of U.S. application Ser.No. 15/043,526 filed Feb. 13, 2016; which is a continuation in part ofU.S. application Ser. No. 14/822,447 filed Aug. 10, 2015; all of whichare incorporated by reference herein.

The specification relates to illuminating a region that has areas not ornot easily visible to the human eye and highlighting those areas of theregion with a visible light overlay, and in particular for using thecapability for surgical applications.

For many processes, including surgical operations, areas of interest ina region may not be visible to the eye of the person performing theprocess, but may be detectable with an imaging device. For instance,fluorescence can be used to identify areas of a region including areasof surgical interest. Some materials may exhibit fluorescence atnon-visible wavelengths. Other areas of interest may exhibit too low acontrast to the human eye to be easily visible. For these situations,which include some parts of the human body, detecting non visible areasof interest and highlighting them visibly may be desirable.

BRIEF DESCRIPTION

In some embodiments, devices and methods may be provided to image aregion with a suitable imager capable of detecting areas of interest ofthe region either not or not easily discernible to the human eye. All orpart of the non-visible image scene may be overlaid visibly back ontothe imaged region with visible light, to highlight the areas of interestdetected from the acquired images.

In some embodiments a device for visibly highlighting areas of a regionmay be provided including an imager configured to image the region witha sensitivity to at least one of wavelength, light level, or contrastgreater than the human eye, an overlay element configured to visiblyhighlight areas of the region and registered to the imager to producealignment of imaged features with highlighted features at the samelocation on the region, and at least one of a controller executing aprogram or logic configured to process acquired images from the imagerto identify areas of the region determined not visible to the human eye,and control the overlay element to visibly highlight those areas on theregion.

In some embodiments a method for visibly highlighting areas of a regionmay be provided including imaging the region with an imager withsensitivity to at least one of wavelength, light level, or contrastgreater than the human eye, highlighting visibly areas of the regionregistered to the imager to produce alignment of imaged features withhighlighted features at the same location on the region, and processingacquired images from the imager to identify areas of the regiondetermined not visible to the human eye, and control the illuminator tovisibly highlight those areas on the region.

In some embodiments an illuminator configured to illuminate the imagedregion may be employed.

In some embodiments the imager may be sensitive to wavelengths outsidethe visible range.

In some embodiments the illuminator and the imager may both operate atwavelengths outside the visible range.

In some embodiments the illumination may be modulated and the imager maybe configured to capture images synchronized to the modulation, allowingfor immunity to ambient light.

In some embodiments the imager, illuminator, and overlay element may beconfigured as one unit at one working distance.

In some embodiments a relationship between the imager and illuminatorwavelengths may include being one of different wavelengths, overlappingwavelengths or the same wavelengths.

In some embodiments the highlighting may be configured to be at leastone of a single visible color or multiple colors, selected for highcontrast with the colors of the region of interest.

In some embodiments the processing may include thresholding the image tohighlight the region of interest where the thresholding includes atleast one of a predetermined intensity level, predetermined intensityvariance, or ratio of intensities at 2 or more locations, 2 or morewavelengths, or at 2 or more locations of full width half max signal.

In some embodiments, the overlay element comprises one or more of astandard light source projector, a laser scanner controlled by thecontroller to project the visible image by scanning the image on theregion, or one or more visible LED's actuated by the controller to scanthe image on the region.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects and advantages of the embodiments provided herein are describedwith reference to the following detailed description in conjunction withthe accompanying drawings. Throughout the drawings, reference numbersmay be re-used to indicate correspondence between referenced elements.The drawings are provided to illustrate example embodiments describedherein and are not intended to limit the scope of the disclosure.

FIG. 1 is an overview of overlay system used in particular medicalapplication according to illustrative embodiments;

FIG. 2 is a block diagram of an general illustrative device embodiment;

FIG. 3 is a block diagram of another general illustrative deviceembodiment;

FIG. 4 is a block diagram of a more detailed illustrative deviceembodiment;

FIG. 5 is a block diagram of a specific illustrative device embodiment;

FIG. 6 is a flowchart of a illumination synchronization processaccording to an illustrative embodiment

FIG. 7 is flow chart of a method according to an illustrativeembodiment;

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

One or more embodiments described herein may provide for a visiblehighlighting of features of interest in a region not visible to the eyeidentified from an images of those features acquired by an imager withmore sensitivity to the features than the human eye.

One or more embodiments described herein may provide for visiblyhighlighting features of the human body for surgery.

One or more embodiments described herein may provide for visiblyhighlighting features for surgery by imaging areas of interest thatfluoresce in a non-visible wavelength and projecting visiblehighlighting back onto those features.

One or more embodiments may include feature identification andhighlighting that is not sensitive to ambient light such as operatingroom lighting.

Various aspects of the embodiments may include any combination ofprocessing elements and memory that may include computing devicesexecuting software routines, such devices including computers andPersonal Electronic Devices, as well as programmable electronics, logiccircuits and other electronic implementations. Various combinations ofoptical elements may be employed including lasers, LED's and other lightsources, filters, lenses, mirrors, beamsplitters and the like. Thedetails of the optical, electronic, and processing embodiments describedherein are illustrative and are not intended as limiting as alternativeapproaches using other combinations of like elements may be used toaccomplish the same results in essentially the same manner

A method for discriminating parathyroid material from thyroid material,using autofluorescence, is described in U.S. patent application Ser. No.13/065,469. This application has inventors in common with the currentapplication, and is incorporated by reference in its entirety. Thisapplication discloses that when exposed to radiation in a narrow bandabout 785 nm, which is just outside visible range, both the thyroid andthe parathyroid auto-fluoresce in a wavelength range above 800 nm, alsonot visible, sometimes centered at 822 nm, and that the parathyroidregions fluorescence intensity is significantly higher than the thyroidregions. This effect can be used to discriminate between the two areasfor surgery, for even though the general locations of parathyroid tissueare known, they are hard to discriminate visually accurately enough forsurgery, which can be a problem with parathyroid surgery. One detailedembodiment disclosed herein may be applicable to parathyroid surgery. Asshown in FIG. 1, an optical overlay device 1 according to an embodimentof the current disclosure may be used to visibly highlight theparathyroid regions of a patient's exposed internal neck region duringsurgery. Systems and methods particularly suitable to fluorescenceapplication such as the parathyroid example are described in commonlyowned U.S. application Ser. No. 14/822,477, filed Aug. 10, 2015, whichis incorporated by reference in its entirety.

The parathyroid application is an example of a process where anoperator, in the parathyroid case, a surgeon, needs to perform a processon areas of interest in a region where he may not be able to see theareas of interest by eye. In the case of the parathyroid example, thefluorescing parathyroid material is fluorescing in the near infrared.However in general, imagers of various types can be made that exceed theeye's sensitivity not just in terms of detectable wavelengths, but alsoin terms of contrast or low light sensitivity.

FIG. 2 illustrates a general embodiment of a system 1 for overlayingvisible highlighting onto non-visible areas 9 of interest in a region.Optionally the region may be illuminated by an illuminator 2. For thecase of fluorescence, this illuminator may be selected to operate at awavelength which will stimulate the desired fluorescing behavior. Or theilluminator could be a visible light source, such as an operating tablelight or other workstation lighting device. It could be a UV light, oranother EM radiation source. Not all applications will require theilluminator. Imager 7 is selected to be sensitive to the characteristicsof the area of interest, and may be an IR imager, a UV imager, an X-rayimager, other electromagnetic imager, a low light visible imager, or avisible imager configured either in hardware or with signal processingfor high contrast imaging. The imager of whatever type should be able todetectably image features the operator cannot see by eye. The imager andoptional illuminator may either or both operate at non-visiblewavelengths, and the imager and optional illuminator may or may notoverlap in operating wavelength. Processor 8 receives the imagesacquired by imager 7 and processes the images to identify the areas ofinterest by filtering or processing of various types. Filtering mayinclude, high/low intensity, deviation from average, within a range,near predetermined locations, or any combination thereof, or otherfiltering/processing techniques. The processing may include thresholdingthe image to highlight the region of interest where the thresholdingincludes at least one of a predetermined intensity level, predeterminedintensity variance, or ratio of intensities at 2 locations on theregion, the ratio of 2 wavelengths, or the ratio data representative ofthe full width half max of signal at two or more locations, wavelengthsor the like. Other processing techniques for feature identification maybe suitable. The desired result is identification locations on theimaged region likely to be the material of interest. The processor thencontrols an overlay element 6 to highlight the areas 9 identified by thefiltering on the region with visible light highlighting visible to theoperator. The overlay element and imager are registered to the workingregion by mechanical, optical, image processing or some combination oftechniques. For embodiments where the imager, overlay element andoptional illuminator are configured as one or more units all at oneworking distance, such registration is straightforward.

A variety of overlay techniques could be employed in variousembodiments. A standard light source projector could be used withsuitable optics. Other embodiments could use a scanning laser as used inLaser show devices, and other illumination applications, by inputtingthe signal from the camera a 7 and programming the projector laser 6output to accurately illuminate specific areas corresponding areas ofinterest. Other embodiments could employ 1 or more, and up to 4 or moreLEDs/Lasers that are controlled by X and Y axis motors (possibly withthe laser at a hinge point) to accomplish the same result as a laserscanner. For this embodiment, programming the motor movements toilluminate a specific point with the laser may indicate where the areasof interest lies. For the parathyroid case, there are four parathyroids,the system could either quickly move the motors of one LED/Laser toilluminate multiple areas with sufficient refresh rate or could controlup to 4 LEDs/Lasers to independently accomplish that same function. Thevisible overlay color may be single color or multi-color and chosen tocontrast with the background. For instance for surgical operations, agreen overlay color contrasts well with the predominantly redbackground.

FIG. 3 illustrates the case where the illumination 2 is in the near IR,the imager 7 is an IR camera and the overlay element 6 is a visibleprojector. This is useful for cases such as the parathyroid where areasof interest either fluoresce or are otherwise more visible in the ir.For many surgical applications the working region containing the areasof interest is usually less than 10 cm in the longest direction.

FIG. 4 shows a more detailed illustrative device embodiment for the IRfluorescence overlay 1. Light source 2 at a first wavelength bandwidthilluminates a region of interest 9. The first wavelength bandwidth isused to stimulate emissions or fluorescence at a second wavelengthbandwidth, which may be different from the first, expected from areas tobe identified of the region of interest. Those emission wavelengths arenot visible for applications utilizing the device. An optional filter 4may be used to pass wavelengths within the emission bandwidth and blockothers. An optional lens may be used 5 to set working distance of thedevice. Camera 7 is chosen to be capable of imaging the emissionwavelength bandwidth or at least the portion passed through the filter4. The image is acquired by computing device/logic 8 which also controlsa visible light projector 6. Controllable Projector 6 and camera 7 areregistered such that the imaged area and the projected area are alignedboth in orientation and size so that features in the camera image or anyportion of the camera image project back down on the region 9 such thattheir visible projection aligns precisely onto the actual physicalfeatures. The registration may be accomplished through optical design,which may be improved using calibration regions with definable edges andprogramming the projector to match such calibration pieces at a desiredworking distance for actual operation. Such edge or other featuredetection may be updated in actual use by observing and correlatingimage features in actual regions of interest. The projection may be madeco-linear with the imaging axis by use of a partially reflective element3, such as a beamsplitter. Obviously different optical arrangements,such as which elements are on or off axis, may be accomplished withdifferent arrangements of optical elements and still function asdescribed for the illustrative arrangement of FIG. 4.

In FIG. 5 a specific embodiment aimed at the parathyroid surgicalapplication is Acquisition parameters needed for clinical use, suggestan exemplary set of design criteria for an overlay device−Spot size=5 cmdiameter, Image size=5×7 cm, Lateral resolution=500 microns, SNR=4 (forlaser power density=7 mW), Image acquisition time=100 ms.

For the parathyroid surgical overlay device of FIG. 6 a projector-based6 system is employed to enhance visualization by overlaying NIRautofluorescence structures with visible light in real-time. Thesurgical field 9 is illuminated with a near-infrared (NIR) diode laser 2(λ=785 nm, Innovative Photonics Solutions) coupled through a 400-μmfiber optic patch cable, filtered with a laser clean-up filter(Semrock), and focused with a plano-convex lens (f=75 mm, Thorlabs) toachieve a 5-cm diameter spot size. The emitted NIR autofluorescence isfiltered through a long-pass filter (800-nm, Semrock) to eliminate theelastically scattered light and specular reflectance and detected usinga NIR CMOS camera 7 (Basler AG). The raw NIR image is sent to aprocessor including an image processing unit 8 for real-time imageprocessing. The processed image is then be sent to a high-lumenlight-emitting diode (LED) projector 6 (AAXA Technologies), whichprojects the false colored image through a dichroic filter/mirror 3(801-nm, Semrock) and on to the surgical field 9.

A software program is used for real-time processing of the NIRfluorescence images. Images collected by the NIR camera is sent to theprocessor. Exemplary Image Processing/filtering includes initiallysubtracting a dark frame. The image is sent through a convolution filterand a feature extraction algorithm to achieve morphologicaltransformation. This series of steps achieves maximal signal intensityin the areas of low level fluorescence present in the surgical field.The processed image is sent to the projector and converted to a greencolor intensity before projection on the tissue. The software alsoallows co-registration of the projected image with real space using aseries of x and y manipulation controls. Image coregistration may beperformed prior to each imaging event using a fluorescent grid phantomas a registration target.

Spatial resolution of the parathyroid overlay system of FIG. 5 should beable to resolve features as small as 3 mm parathyroid glands. The systemshould be capable of detecting fluorescence with 30 mW excitation at 785nm thus qualifying for class 3R medical device status.

NIR illumination/excitation in the 30 mW range may be accomplished witha laser. However the use of lasers may complicate meeting FDAregulations for application such as operating room equipment and alsomay the flexibility of use in the operating room (i.e. requiring a footswitch to activate the laser). LEDs excitation sources set to the samewavelength and similar power may be suitable and in some cases desirablealternatives. A LED based device tested using LEDs from Epitex(SMBB780D-1100-02) has shown the ability of the LED(s) to produce enoughradiant power to a parathyroid surgical site at a working distance of500 mm away from the surgical to detect the fluorescence signals fromparathyroid and thyroid. The performance of such a system may becomparable with a diode laser based device and may have advantages asdescribed above. A decision will be made at this point about whether tocontinue using a laser or to switch to using LEDs. An LED illuminatormay be realized as a series of LEDs at the annular opening around animager camera lens so as to increase the radiant output as well as toeliminate or reduce any shadowing effect that is cause when using oneLED and working with a surgical site that isn't flat.

It is desirable for many applications to be able to operate the overlaysystem in whatever suitable ambient lighting exists. For instance,surgical operating rooms often include bright light sources that mayinterfere with the overlay device, particularly when the wavelengths ofinterest are NIR. It may be inconvenient however to turn off ambientlighting. For example, for a surgical application:

-   -   a. The surgeon and staff cannot adequately see if some or all        lights are off    -   b. Even if most lights are off during signal acquisition,        sources of light such as indicator lights, monitors etc. may        still leak through. The leaked light main spectral components in        the same spectral range as the wavelengths of interest. This        results in decreased signal-to-noise ratio and decreases the        parathyroid detection sensitivity.

To use the overlay system with ambient lighting on, it may be desirableto employ modulated illumination to synchronize image acquisition withthe modulation characteristic. For example, a simulated lock-in/FastFourier Transfer (FFT) with an imaging sensor to eliminate light leakageinto the sensor is one approach. This light leakage can result in highambient noise levels that can exceed the output from detected features,for example autofluorescence. By utilizing the process described below,the fluorescence signal intensity will remain visible while eliminatingany other light that can leak into the sensor without having to dim theambient lighting such as Operating Room (OR) lights. Theillumination/excitation light (laser or LED or similar) is modulated bypulsing at a frequency that is distinct from those of the ambientlighting. The resulting features of interest will emit at that samefrequency. As such, by taking the FFT of the acquired image, the desiredoutput (such as from autofluorescence) can be differentiated from anyother light. FIG. 6 is a flow chart of an illustrative process. In step60, the illuminator is modulated at a predetermined frequency. In step61 image data is acquired from a sensor. In step 62 groups of pixels aresorted into bins. In step 62 a Fast Fourier Transform (FFT) is executedon the bins. In step 64. modulation frequency synchronized image data isused to develop overlaying

An alternative approach to the use of FFT is to acquire a diffusereflectance image with ambient lights on and illuminator off. This imagecan then be used to subtract the background and along with imageprocessing can be used to obtain a high-contrast fluorescence image withthe light on. Electronic lock-in approaches may also be suitable.Another alternative embodiment may use image detection technology thatutilizes a Gabor filter, which is used for edge detection. Frequency andorientation representations of Gabor filters are similar to those of thehuman visual system, and they have been found to be particularlyappropriate for texture representation and discrimination. In thespatial domain, a 2D Gabor filter is a Gaussian kernel functionmodulated by a sinusoidal plane wave. Another embodiment may includespecific lighting (with defined wavelength) that is part of the camerasystem chosen to lessen the detrimental external lighting effects on theoverlay system. This lighting may be cropped out using the filters infront of the camera, or may source from a limited bandwidth illuminatorsuch as specific LED's or laser illuminators. Thus the OR lights may beturned off with exception of these specific lights that are providedwith the camera system to illuminate the desired region.

FIG. 7 is a flow chart of a method of operating a general overlay devicesuch as the one shown in FIG. 2 for a case where a non-visiblefluorescence of a material of interest may be used to identify thelocations of that material in a region. In step 70 a region isoptionally illuminated with electromagnetic radiation, which may be anyof a variety of types, UV, bright visible, low intensity visible, X-Ray,other EM radiation, narrowband, at a wavelength chosen to excite adesired non-visible fluorescence of a material of interest, or any typechosen to aid in identifying features not discernible by eye. In step 71one or more images are acquired in a bandwidth that is useful fordetecting areas of interest. In step 72 the image is filtered to selectidentify the non-visible areas of interest. Filtering may include,high/low intensity, deviation from average, within a range, nearpredetermined locations, or any combination thereof, or otherfiltering/processing techniques. The desired result is identifyinglocations likely to be the material of interest. In step 73, the overlayelement is controlled to highlight all or part of the acquired image,which may be just of the selected locations, back onto the region withvisible light. This will have the effect of illuminating the selectedlocations 9 on the region with visible highlighting.

The embodiments described herein are exemplary. Modifications,rearrangements, substitute devices, processes etc. may be made to theseembodiments and still be encompassed within the teachings set forthherein.

The invention claimed is:
 1. A system for distinguishing between thyroid tissue and parathyroid tissue of a patient during surgery, the system comprising: an emitter configured to direct radiation having a wavelength of approximately 785 nm toward a neck portion of the patient to illuminate and stimulate the thyroid tissue and the parathyroid tissue into autofluorescence; an imager configured to collect radiation corresponding to the autofluorescence of the thyroid tissue and the parathyroid tissue having wavelengths above approximately 800 nm; at least one processor for processing images from the imager, the at least one processor configured to distinguish between the radiation from the autofluorescence of the thyroid tissue and the radiation from the autofluorescence of the parathyroid tissue based on autofluorescence intensity; and an overlay device controlled by the at least one processor, the overlay device configured to visibly highlight, in the neck portion of the patient, areas of the autofluorescence of the parathyroid tissue to facilitate performance of parathyroid surgery.
 2. The system of claim 1, wherein the emitter is a UV light emitter having an excitation level of 30 mW.
 3. The system of claim 1, wherein the overlay device is configured to visibly highlight features as small as 3 mm parathyroid glands.
 4. The system of claim 1, wherein the overlay device includes a laser scanner controlled by the at least one processor to visibly highlight portions of four parathyroid glands of the patient.
 5. The system of claim 1, wherein the overlay device includes one or more LEDs and/or lasers controlled by the at least one processor to visibly highlight portions of four parathyroid glands of the patient.
 6. The system of claim 5, further comprising X and Y axis motors controlled by the at least one processor for controlling orientation directions of the one or more LEDs and/or lasers.
 7. The system of claim 1, wherein the illumination by the emitter is modulated, and wherein the imager is configured to capture images synchronized to the modulation allowing for immunity of the imager to ambient light.
 8. The system of claim 7, wherein synchronization of the imager to the modulation serves to eliminate adverse effects of light leakage.
 9. The system of claim 1, wherein the emitter, the imager, and the overlay device are configured as one unit at one working distance from the neck portion of the patient.
 10. The system of claim 1, wherein the visibly highlighted areas are visibly highlighted by the overlay device with at least one color selected for high contrast with of the neck region of the patient.
 11. A method for distinguishing between thyroid tissue and parathyroid tissue in a neck portion of a patient during surgery, the method comprising: directing radiation having a wavelength of approximately 785 nm toward the neck portion of the patient to illuminate and stimulate the thyroid tissue and the parathyroid tissue into autofluorescence; collecting radiation corresponding to the autofluorescence of the thyroid tissue and the autofluorescence of the parathyroid tissue having wavelengths above approximately 800 nm; distinguishing between the radiation from the autofluorescence of the thyroid tissue and the autofluorescence of the parathyroid tissue based on autofluorescence intensity; and visibly highlighting, in the neck portion of the patient, areas of the autofluorescence of the parathyroid tissue to facilitate performing parathyroid surgery.
 12. The method of claim 11, wherein the radiation is directed from a UV light emitter having an excitation level of 30 mW.
 13. The method of claim 11, wherein the visibly highlighting includes visibly highlighting features as small as 3 mm parathyroid glands.
 14. The method of claim 11, wherein the visibly highlighting is performed by a laser scanner.
 15. The method of claim 11, wherein the visibly highlighting is performed by one or more LEDs and/or lasers.
 16. The method of claim 15, wherein X and Y axis motors control orientation directions of the one or more LEDs and/or lasers.
 17. The method of claim 11, wherein the illumination is modulated, and the collecting radiation includes capturing images synchronized to the modulation allowing for immunity of the collecting radiation to ambient light.
 18. The method of claim 11, wherein the directing radiation, the collecting radiation, and the visibly highlighting are performed by one unit at one working distance from the neck portion of the patient.
 19. The method of claim 11, wherein the visibly highlighting includes visibly highlighting with at least one color selected for high contrast with of the neck region of the patient.
 20. A method for distinguishing between thyroid tissue and parathyroid tissue of a patient during surgery, the method comprising: directing radiation from an emitter having a wavelength of approximately 785 nm toward the neck portion of the patient to illuminate and stimulate the thyroid tissue and the parathyroid tissue into autofluorescence; collecting radiation corresponding to the autofluorescence of the thyroid tissue and the autofluorescence of the parathyroid tissue having wavelengths above approximately 800 nm using an imager; processing images from the imager using at least one processor to distinguish between the radiation from the autofluorescence of the thyroid tissue and the autofluorescence of the parathyroid tissue based on autofluorescence intensity; controlling an overlay device including one of a laser scanner, one or more LEDs, or one or more lasers to visibly highlight the parathyroid tissue in the neck portion of the patient to facilitate performing parathyroid surgery. 